Improving component selection and monitoring with controlled experimentation and automated measurements

Please cite this article in press as: C. Becker, A. measurements, Inform. Softw. Technol. (2010), Context: Anumber of approaches have been proposed for the general problemof software component evaluation and selection. Most approaches come from the field of Component-Based Software Development (CBSD), tackle the problem of Commercial-off-the-shelf component selection and use goal-oriented requirements modelling and multi-criteria decision making techniques. Evaluation of the suitability of components is carried out largely manually and partly relies on subjective judgement. However, in dynamic, distributed environmentswithhighdemands for transparent selectionprocesses leading to trustworthy, auditable decisions, subjective judgements and vendor claims are not considered sufficient. Furthermore, continuousmonitoring and re-evaluation of components after integration is sometimes needed. Objective: This paper describes how an evidence-based approach to component evaluation can improve repeatability and reproducibility of component selection under the following conditions: (1) Functional homogeneity of candidate components and (2) High number of components and selection problem instances. Method: Our evaluation and selection method and tool empirically evaluate candidate components in controlled experiments by applying automated measurements. By analysing the differences to systemdevelopment-oriented scenarios, the paper shows how the process of utility analysis can be tailored to fit theproblemspace, anddescribes amethodgeared towards automated evaluation in anempirical setting. We describe tool support and a framework for automated measurements. We further present a taxonomy of decision criteria for the described scenario and discuss the data collection means needed for each category of criteria. Results:To evaluate our approach,wediscuss a series of case studies in the area of digital preservation.We analyse the criteria defined in these case studies, classify them according to the taxonomy, and discuss the quantitative coverage of automated measurements. Conclusion: The results of the analysis show that an automated measurement, evaluation and selection framework is necessary and feasible to ensure trusted and repeatable decisions. 2010 Elsevier B.V. All rights reserved.